Saturated Transformers are Constant-Depth Threshold Circuits

Abstract Transformers have become a standard neural network architecture for many NLP problems, motivating theoretical analysis of their power in terms formal languages. Recent work has shown that transformers with hard attention are quite limited (Hahn, 2020), as they can be simulated by constant-depth AND/OR circuits (Hao et al., 2022). However, is strong assumption, which may complicate the relevance these results practice. In this work, we analyze circuit complexity saturated attention: generalization more closely captures patterns learnable practical transformers. We first show transcend known limitations hard-attention then prove floating-point values threshold circuits, giving class TC0 an upper bound on languages recognize.